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ABSTRACT 

Digital learning is becoming a prevalent everyday human behavior. Effective digital learning services are integral for 
educational innovation and constitute competitive advantages for education businesses. Quality management in 
e-learning research and development is thus of utmost importance and needs both strong conceptual and empirical 
underpinnings. On the one hand, this work delineates a new model that conceptualizes main critical issues in e-leaming 
research and development projects. The model will foster substantial progress in evaluation and theory development of 
digital learning. On the other hand, the evaluation of how central features of digital learning services affect learning is 
indispensable. Here, empirical studies will investigate three key stages of learning: the initial situation, the learning 
procedure, and the assessment procedure. With regard to the initial situation, how do existing learner characteristics relate 
to learning performance in digital learning modules? Second, how do visual features of the digital learning content affect 
the visual attention, cognitive load, and learning performance of the users? Third, how do additional information in 
textual feedback affect the learning performance in digital multiple-choice quizzes? Overall, this work presents a new 
model for the systematic evaluation of e-learning and investigates key learning stages to illustrate the importance of 
developing evidence-based guidelines. Turning both approaches into common practice in e-leaming research and 
development will substantiate theory development, quality management, and project success. 
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1. INTRODUCTION 

Many educational institutions offer electronic services for learning (e-learning) and teaching (e-teaching), 
e.g., to improve education, to develop new learning spaces, to enhance mobility, to foster international 
outreach, or to gain more evidence on the quality of teaching and learning processes. Common arguments for 
choosing digital learning services (i.e., products of e-learning projects) can be found in various existing 
guidelines on e-learning of educational institutions and companies. On the one hand, these guidelines often 
list pros and cons of digital learning services per se (e.g., more flexibility and less social interaction, 
respectively). However, neither does this approach enable to explain failure or success, nor does it consider 
that e-learning projects are often very different. Thus, including the pros and excluding the cons does not 
appear to be a satisfying heuristic for e-learning project teams. On the other hand, building new projects 
based on best practices is also not necessarily a promising approach. Some of these practices might include 
insights that highly depend on contextual or unknown factors. Obviously, one should not replicate successful 
concepts in different contexts, e.g., in higher education and in high schools, without adapting to the context. 
Still, it will remain less clear how to handle missing information during adaptation, e.g., if best practices do 
not explain how the teams decided on implementing specific features in digital learning services. Project 
teams and researchers need such essential information to determine and to discuss educational effectiveness 
and efficiency. In turn, e-learning project teams could clarify promises and pitfalls of digital learning services 
by providing comprehensive documentations including empirical evidence based on scientific inquiry. 

First, only if e-learning project teams continuously monitor and explicitly document the quality of their 
project, they will be able to revise the route of the project when necessary. To monitor the quality of 
e-learning projects, they may adapt evaluation criteria to some degree, e.g., from international standard 
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frameworks on e-learning quality management (ISO/IEC DIS 40180), on human-centered design for 
interactive systems (ISO 9241-210), or on usability (ISO 9241-11). However, they should carefully integrate 
such standards with systematic evaluation strategies and frameworks that consider features specific to 
e-learning projects. Digital learning services become increasingly available and their use may become 
necessary in realizing some contexts of learning. Consequently, this work formulates conceptual solutions to 
current critical problems with regard to the evaluation of e-learning. 

Second, e-learning projects fail or succeed depending on how much the resulting digital learning services 
are used (i.e., users will usually accept or decline them) and depending on how well they enable learning. 
Hence, e-learning project teams should aim for an entire understanding of the target audience from the very 
start. Does the digital learning service satisfy the users’ needs? Does it facilitate their learning process? What 
do the users actually learn? The project teams need to evaluate their products based on empirical 
investigations to answer such essential questions. Only then, they can formulate valid statements on how 
their service affects learning and teaching in comparison with other (technology-enhanced) implementations. 
In line with the aforementioned three questions, this work focuses on particular effects during three key 
stages of common learning procedures. First, the initial stage of learning, where users bring along prior 
knowledge, motivation, and other learner characteristics. Second, the stage of learning processes, here 
focusing on learners that visually perceive some learning content. Third, the assessment stage, which usually 
follows learning processes to assess the learning performance. Studying these key learning stages intends to 
clarify the role of personality, visual learning behavior, and feedback in digital learning services. E-learning 
project teams that consider such evidence could knowingly adjust their project routes to success. 

Overall, this work aims to enhance the effectiveness and the (understanding of the) user performance in 
digital learning services and endorses that evidence-based evaluations increase the success rate of e-learning 
projects. The following section focuses on theory development in e-learning and on the evaluation of 
e-learning projects. Then, in accordance with the three key learning stages, empirical investigations on 
learner characteristics, visual learning behavior, and learning performance of e-learning users will be 
outlined. 


2. E-LEARNING EVALUATION AND THEORY DEVELOPMENT 

Digital learning services can be used in almost any context of learning. Unique and discipline-specific factors 
are often inherent to many of these contexts. Therefore, the related e-learning projects are often neither easy 
to compare nor to replicate. Unique projects can yield innovative outcomes, but scientific inquiries aiming 
for comprehensive theories of e-learning need a way to accumulate the generated knowledge. This strategic 
deficit hampers progress in e-learning evaluation and theory development. To enhance progress in evaluation 
and theory development, e-learning project teams could consider the eleven critical issues that Rtith and 
Kaspar (2017) formulated in the E-Learning Setting Circle. Accordingly, the next two paragraphs present the 
rationale and the two most important critical issues from that already published part of this work. 

With regard to evaluation, it is important to understand that the assessment of the main goals of e-learning 
projects - to actually, effectively, and efficiently enhance learning - is not a simple task. One reason for this 
is that traditional learning theories - e.g., behaviorism, cognitivism, constructivism, and active theory (cf. 
Pange and Pange, 2011) - are compatible with some approaches of e-learning, but do not fully capture the 
scope of e-learning. The discrepancy between traditional learning and e-learning is due to the fact that 
e-learning projects are highly artificial phenomena (cf. Phillips, Kennedy, and McNaught, 2012). To 
generalize natural phenomena such as gravitation, one can refer to and build on multiple comparable 
experimental evidence. In contrast, e-learning projects are necessarily co-determined by situational factors. 
These range from general boundary conditions to specific, partly inherent aspects of technological processes. 
Situational factors lead to a vast amount of decision routes e-learning project teams might take, which in turn 
complicates comparisons across the projects’ processes and products. Nevertheless, each team needs solid 
arguments that go far beyond method novelty or technological advance to create effective implementations. 
Further, the teams that realize e-learning projects often consist of experts from different disciplines that 
temporarily collaborate. Consequently, performing and documenting decision-making comprehensibly is 
essential, e.g., to allow each team to trace how another team worked out an effective combination of 
technological features. Since project routes typically vary, it is of particular importance that the teams 


228 



International Conference e-Leaming 2017 


carefully monitor which routes are promising for goal achievement - decisions at each stage of the project 
might co-determine its failure or success. During each project, many feedback loops and iterations are 
conceivable and many technological features enable user interaction. Overall, more situational factors 
co-determine e-learning projects further than in case of traditional learning projects. Comprehensible project 
work referencing to theory will hence distinctly facilitate the generalization of e-learning project results. 
Importantly, the most important issues in e-learning projects are to elucidate the role of decision-making and 
to assess the goal attainment level. 

On the one hand, due to the artificial nature of e-learning projects, decision-making is integral to all 
project activities. The project teams always need to explicate in what regard stakeholders, project 
deliverables, and other situational factors drive their decision-making. Importantly, if the influence of 
situational factors is not considered, neither will it be easy to compare evidence from e-learning projects, e.g., 
for benchmarking of project results, nor will it be straightforward to generalize such evidence, e.g., for 
companies or educational institutions aiming to formulate comprehensible best and worst practices. 

On the other hand, the primary goal of e-learning projects - to enhance learning - and other existing goals 
should be defined from the start. Similarly, only if the methodology and quality metrics are clearly 
formulated, e-learning projects will become comparable and will be eligible for benchmarking. 
Consequently, only if project outcomes and project goals are clearly related within and across projects, there 
will be systematic and substantial progress in the area of e-learning research and development. 

Hence, Rtith and Kaspar (2017) defined decision-making and the assessment of the goal attainment level 
as the universal element and the guiding element of the E-Learning Setting Circle, respectively. The 
E-Learning Setting Circle provides a practical solution for all e-learning project teams that aim to enhance 
the validity of their e-learning project evaluations. It is thus essential that e-learning project teams document 
the relevant decision routes and project results meticulously. The scientific purpose of the model is to 
enhance comparability and generalizability of evidence from e-learning projects. In practical terms, those 
teams that apply the model could identify and anticipate decision routes that actually lead to failure or 
success of e-learning projects more systematically. 


3. UNDERSTANDING LEARNING PERFORMANCE IN E-LEARNING 

The empirical part of this work scrutinizes the impact of learner characteristics, visual attention, and 
feedback on learning performance in e-learning. Each of the next three paragraphs only presents the rationale 
of the planned studies. The full methodology will be specified in the prospective publications. 

First, to illustrate the importance of learner characteristics, picture two extreme versions of how digital 
learning could take place. On the one hand, one e-learning project team decides to implement one 
standardized learning path that forces each learner to complete the same learning module. One the other 
hand, another team decides to investigate differences between their users first to then deliver the learning 
content in a customized way. Since uniform and customized digital learning services provide different 
degrees of freedom, the users might show different levels of learner motivation (see, e.g., Keller, 2016; 
Keller and Suzuki, 2004) and self-regulation (see, e.g., Liaw and Huang, 2013). Nakayama, Mutsuura, and 
Yamamoto (2014) highlighted the important relation between personality and learning performance in an 
online course. Koster et al. (2015) showed that users recognized personalized content more often than 
non-personalized content in the context of digital online advertisements. Accordingly, this study tests if 
digital learning services tailored to specific personal learner characteristics improve learner motivation and 
learning performance. In an experimental setup, the users will complete a digital learning sequence at their 
own speed. The learning modules will differ with respect to customization, i.e., the degree to which the 
learning content meets the users’ needs. To assess the memory performance of the learners, they will 
complete computer-based tests constructively aligned to the learning content. Computer-based questionnaires 
will quantify the learner motivation. Main demographical characteristics (e.g., age and gender) and the 
learners’ expectations (Paechter, Maier, and Macher, 2010) will be measured alike and will be included as 
covariates in the data analysis. The results will show if personalized digital learning content increases or 
decreases the learner motivation and the learning performance. Potential pitfalls of personalized learning will 
be discussed, e.g., if customized learning reduces the learners’ opportunities to learn how to learn. In 
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practical terms, the results of this study might help to explain when customization of digital learning services 
could enhance learning performance in general. 

Second, with regard to the learning procedure, it is specifically important to evaluate how visual features 
of the learning content (e.g., textual and pictorial multimedia components) affect the learning performance 
and user experience of the learners. There is much evidence that design choices in multimedia content can 
systematically influence visual attention and learning amongst other cognitive mechanisms (for an overview 
see, e.g., Mayer, 2014). First, according to the cognitive theory of multimedia learning (cf. Mayer, 2014), 
learners actively process the digital learning content by attending, selecting, organizing, and integrating 
relevant information. Second, they use two distinct channels to process textual and pictorial information. 
Third, learners only process up to some quantity of information at once, because the capacity to learn is 
limited. Nevertheless, particular designs of multimedia learning content might reduce the cognitive load of 
the learners (Mayer and Moreno, 2003). Accordingly, this study will compare how different visual features of 
digital learning content reduce the cognitive load and if they enhance the subsequent learning performance. 
To assess the learning performance, the learners will complete a computer-based test constructively aligned 
to the learning content. First, an online experiment will test for differences in learning performance and will 
allow estimating the effect size. If learners actually perform differently, a second experiment will test for 
similar effects under laboratory conditions. If this is the case, a third experiment will implement the same 
procedure while also assessing the gaze behavior of the learners by means of eye tracking. This enables to 
identify those visual features of the digital learning content that are potentially relevant for the difference in 
learning performance of the users (cf. Rakoczi and Pohl, 2012). In all experiments, the learners will indicate 
their subjective cognitive load based on self-reported difficulty. In the third experiment, objective eye 
tracking measures will also be used to quantify the cognitive load (cf. Zagermann, Pfeil, and Reiterer, 2016). 
In addition, all experiments will test if the subjective evaluation of the digital learning service in terms of 
user experience affects the learning performance. These additional data will increase the validity of the 
measures of visual attention. Together, the related data might indicate how visual features of digital learning 
content elicit beneficial or malign effects on learning performance. 

Third, as to the assessment procedure, digital tools are especially powerful in providing opportunities for 
formative assessments for learning (cf. Brown, 2004) - in contrast to summative assessments of learning 
such as final examinations. Importantly, formative assessments need to include effective feedback. A 
frequently used tool for formative assessments are multiple-choice quizzes. However, the reported testing 
effects in digital quiz-like environments are mixed (Little and Bjork, 2015). To close this gap, three 
subsequent experiments will assess how feedback in digital multiple-choice quizzes affects the learning 
performance. More specifically, it is expected that learners that receive feedback including additional textual 
information outperform learners that only receive information about the correctness of their answers. The aim 
of the first laboratory experiment is to test how the feedback affects immediate learning performance for 
content relevant for an upcoming exam. The second laboratory experiment will focus on the immediate and 
delayed (i.e., one week later) effects of the feedback on learning general knowledge. The third experiment 
will test for feedback effects on learning general knowledge in an unsupervised online setup. Importantly, the 
feedback might also affect metacognition and the subjective evaluation of the quiz. Therefore, all 
experiments will also test for differences in the learners’ response certainty and user experience. Response 
certainty is a metacognitive estimate based on prior knowledge (cf. Kulhavy and Stock, 1989) that 
contributes to an understanding of how feedback is processed. Prior experiments have shown that response 
certainty is related to the duration of feedback reception and learning performance in subsequent tests (e.g., 
Kulhavy and Stock, 1989; Mory, 1994). In sum, these experiments will provide multiple evidence from 
different learning contexts, which facilitates evaluating the effectiveness and efficiency of two feedback 
types frequently used in digital multiple-choice quizzes. 


4. CONCLUSION 

Conceptual and empirical progress in the area of digital learning is crucial to unravel the critical factors that 
make for effective digital learning services. First, conceptual progress in e-learning research and 
development needs progress in theory development. To achieve this, e-learning project teams could foster 
project comparability and generalizability by approaching the critical issues of their projects outlined in the 
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E-Learning Setting Circle. Second, e-learning teams usually aim at improving the learning performance when 
deciding for or against the implementation of particular features in digital learning services. Thus, e-learning 
teams should consider empirical investigations on how these features affect the learning performance before 
implementing them in digital learning services. Overall, systematic conceptual and empirical progress 
substantiates effective digital learning services and shapes the future of learning. 
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